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Abstract 

We prove a strong Symmetry of Information relation for random strings (in the sense of Kolmogorov 
complexity) and establish tight bounds on the amount on nonuniformity that is necessary for extracting 
a string with randomness rate 1 from a single source of randomness. More precisely, as instantiations 
of more general results, we show: 

• For all n-bit random strings x and ?/, x is random conditioned by y if and only if y is random 
conditioned by x; 

• While 0(1) amount of advice regarding the source is not enough for extracting a string with 
randomness rate 1 from a source string with constant random rate, amount of advice is. 

The proofs use Kolmogorov extractors as the main technical device. 

Keywords: Symmetry of information, random strings, randomness extraction, Kolmogorov extrac- 
tors. 

1 Introduction 

Kolmogorov extractors are procedures that increase the Kolmogorov complexity rate of strings and 
sequences. Their explicit study was initiated by Fortnow, Hitchcock, A. Pavan, Vinodchandran and 



Wang [ FHP^OG for the case of finite strings and by Reimann [Rei04j for the case of infinite strings. The 



recent paper [ZimlOj is a survey of this field. 

In this paper, we use Kolmogorov extractors as a conceptual and technical device to derive results in 
two directions that otherwise appear unrelated. 

First, we study the symmetry of information phenomenon for random strings. We show that a 
random string x has essentially no information about a random string y if and only if y has essentially no 
amount of information about x. By "essentially no amount of information," we mean that the amount of 
information is bounded by an absolute constant. Up to this paper, it was only known that if x has only 
a constant amount information about y, then y has O(logn) information about x. Thus we replace the 
O(logn) term by 0(1). 

Secondly, we investigate the amount of non-uniformity that is necessary for randomness extraction 
from one source. It is well-known that randomness extraction from a single source is not possible (if we 
exclude some trivial cases). In fact, as a consequence of a result of Vereshchagin and Vyugin |VV02) . we 
note that obtaining a source with randomness rate 1 from a source with randomness rate, say, 0.99 is not 
possible even if the extractor has access to a constant amount of non-uniform information. In contrast, 
we show that an u;(l) amount of non-uniform information is sufficient for this task. 

We continue with a more detailed discussion of the two types of results and of the technical method 
that we use. 
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[http: / /triton.towson.edu/^mziniand The author is supported in part by NSF grant CCF 1016158. 
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Symmetry of Information. Strictly speaking, a string x of length n is said to be random if its 
Kolmogorov complexity is at least n (i.e., C{x \ n) > n) and is said to be random conditioned by string 
y if its Kolmogorov complexity conditioned by y is at least n (i.e., C{x \ y) > n). Since the Kolmogorov 
complexity depends up to an additive constant on the choice of the universal machine, a relaxed definition 
is more robust. Such a definition is obtained if we require that C{x \ n) (respectively, C{x \ y)) are within 
a constant from n. Formally, for a constant c, we say that x is c-random if C(x | n) > n — c, and x is 
c-random conditioned hj y if C{x \ y) > n — c. 

We prove a strong symmetry-of-information relation for random strings: for any n-bit random strings 
X and y, X is random conditioned by y iff y is random conditioned by x. More exactly, we show the 
following. 

Fact 1 (Informal statement; see Theorem \3.4\ for full statement.) Let x and y be two n-bit strings that 
are c-random and such that x is c-random conditioned by y. Then y is c' -random conditioned by x, where 
c' is a constant that only depends on c. 

This result is a consequence of a new form of the Symmetry of Information Theorem, which is tighter 
than the classical theorem of Kolmogorov and Levin [ZL70] for some types of strings, including the class 
of random strings. This is an important class because most strings are random and because counting 
arguments based on Kolmogorov complexity typically use random strings. Symmetry of information is 
one of the basic principles in information theory. It states that for any two random variables X and Y, 
the information in X about Y is equal to the information in Y about X, i.e., I{X : Y) = I{Y : X), 
where I{X : Y) = H{Y) — H(Y \ X) and H{) is the Shannon entropy. The principle also holds in 
algorithmical information theory, provided we replace equal by approximately equal. More formally, for 
two binary strings x and y, we define I{x : y) = C{y) — C{y \ x). Then, the Symmetry of Information 
Theorem of Kolmogorov and Levin states that \I{x : y) — I{y : x) \ = 0(log \x\ + log \y\). For the general 
case of arbitrary strings, the logarithmical term cannot be avoided because there are n-bit strings x and 
y such that |/(x : y) — I{y : x)| > logn — 0(1) (for example, such strings can be obtained by a simple 
modification of Example 2.27 in [LV93j ) . 

Symmetry of information follows immediately from another basic result, the Chain Rule: C{xy) ~ 
C{y) + C{x I y). While it is easy to see that C{xy) < C{y) + C{x \ y) + C^'^\y) 0, the more difficult 
direction in the proof of the chain rule shows that 

C{xy)>C{y) + C{x\y)-C^^\xy). 

We prove a new form of the Chain Rule (which implies the alluded new form of the Symmetry of 
Information Theorem; here and in the rest of this paper, I{x : y) = C{y \ n) — C{y \ x).) 

Fact 2 (Informal statement; see Theorem \3. 1\ for full statement) Let x and y be n-bit strings such that 
C{x\ y) = r2(logn) and C{y\ x) = il(logn). Then 

C{xy \n) > C{x \ n) + C{y \ x) — log/(x : y) 

-0{C^^\x I n) + C(2)(y I n) + C7(2)(y I x)). 

Note that if x is random and y is random even conditioned by x, then C^'^\x \ n) = 0(1), C^'^\y \ 
n) = 0(1), C^'^\y I x) = 0(1), and I{x : y) = 0(1). In this case we obtain C{xy \ n) > C{x \ n) + C{y \ 
x) — 0(1) =2n — 0(1). Fact [1] follows easily from this relation. 

Randomness extraction with small advice. By and large, randomness extraction is an algo- 
rithmical process that constructs a source of randomness of high quality from one or several sources of 

'We use the following shortcut notations: C'^'^\x) is C{C{x)), C'^'^\x \ y) is C{C{x \ y)), C^'^\x \ n) is C{C{x \ n) \ n). 
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lower quality. If we restrict to the case when there is only one input source, one wants to design an 
effective transformation E from the set of n-bit strings to the set of m-bit strings such that for any source 
X "with randomness < k" , E{x) has randomness ~ m. It is desirable to have m k, k (i.e., to extract 
all, or almost all, of the randomness in the source). The problem of randomness extraction has been 
modeled in two ways. In the first model, a source is a probability distribution X over {0, 1}" and its 
randomness is given by the min-entropy Hoo{X). In the second model, a source is a string x G {0, l}" 
and its randomness is given by its Kolmogorov complexity C{x). For our study, the second model is more 
convenient; moreover all the results can be translated in the first model. (For the relation between the 
two models, see [FHP+06] . |HPV09j and also the survey paper [ZimlOj l. 



Formally (in the second model), given the parameter k < n, one would like to have a function 
E : {0,1}"" — >• {0,1}™ such that C{E{x)) ~ m whenever C{x) > k. It is well known that no such 
computable function E exists for non-trivial parameters. Indeed for any given E, consider the string 
y G {0, l}™" with the largest number of preimages. Then C{y \ n) = 0(1) and among its at least 2"~"^ 
preimages there must be some x with C{x) > n — m. In other words, for any given E, there are some 
strings (such as the above x) on which E fails. Thus, in order for a function E to extract randomness 
from any source x with randomness > k, E must have some additional information ax, which we call 
advice about the source. The question is how much such advice information should be provided. 

Fortnow et al. [FHP"'"06] have shown that a constant number of advice bits are sufficient if one settles 
to extracting from strings with linear randomness a string whose randomness rate is 1— e.H More precisely, 
they show that for any positive rational numbers a and e, there exists a polynomial-time computable 
function E and a constant h such that for any x £ {0, 1}" with C{x) > an, there exists a string of 
length h such that C{E(x, Ux)) > (1 — e)m and ni > cn, for some constant c that depends on a and e. 
Note that this result implies that it is possible to construct in polynomial time a list with 2^ strings and 
one of them is guaranteed to have Kolmogorov complexity at least (1 — e)m. 

The shortcoming of Fortnow et al's result is that the randomness rate of the output is not 1. It would be 
desirable that C{E{x, oix)) > m — o{m). We first remark that as a consequence of a result of Vereshchagin 
and Vyugin |VV02] . randomness rate 1 cannot be obtained with a constant number of bits of advice about 
the input. Indeed, we show that if a computable function E : {0, 1}" x {0, 1}'' — )• {0, 1}™ has the property 
that for all strings x with C(x) > an, it holds that there exists ax such that C{E{x,ax)) > (1 — e)m, 
then e > 2/i'+i1i ~ (provided that m = uj{logn + h)). 

In contrast with the above impossibility result, we show that from sources with a linear amount of 
randomness, one can extract a string with randomness rate 1 with basically any non-constant amount 
of advice, such as, for example, the inverse of the Ackerman function. This is an instantiation of the 
following more general result. 

Fact 3 (Informal statement; see Theorem \4-(^ for full statement.) For any m computable from n there 
exists a computable function E with the following property: 

For every n-bit string x with complexity > m, there exists a string ax of length uj{log ^) such that 
C{E{x,ax)) = m — o{m) and the length of C{E{x,ax)) ism. 

Note that the function E from Fact [3] is computable, but no complexity bound is claimed for it. We 
can obtain an extractor E computable by a polynomial-size circuit with almost all the properties from 
Theorem [3l Basically the weakening is that the output length m has to be < cn, for some positive 
constant c. Moreover the polynomial-size circuit is itself computable, in the sense that there exists an 
algorithm that on input n outputs the description of the circuit that computes E. We call such a circuit 
an effectively constructible circuit. 



^The randomness rate of an n-bit string x is C{x)/n. 
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Fact 4 (Informal statement; see Theorem \4.8\ for full statement.) There exists a constant c such that for 
any m computable in polynomial time from n and m < cn there exists a function E, which is computable 
by a polynomial- size effectively constructible circuit with the following property: 

For every n-bit string x with complexity > m, there exists a string ax of length a;(log ^) such that 
C{E{x,ax)) = rn — o{m) and the length of E{x,ax) ism. 

Discussion of technical aspects. We present the main ideas in tiie proofs of Fact [21 Fact O and 
Factm As mentioned, the proofs rely on Kolmogorov extractors. A Kolmogorov extractor is a computable 
ensemble of functions E : {0, x {0, l}"^ ^ {0, 1}™ such that for ah x € {0, and y G {0, l}"^ 
that have Kolmogorov complexity above a certain threshold value and that are sufficiently independent 
(which roughly means that C{y \ x) ^ C{y)), it holds that C{E{x,y)) k, m. 

The general idea for the Chain Rule in Fact [2] is to construct a Kolmogorov extractor E : {0, 1}"" x 
{0,1}" —7- {0,1}™ that extracts almost all the randomness from its inputs, i.e., C{E{x,y) \ n) > C{x \ 
n) + C{y I x) — (small term). If E would be a computable function, then C{E(x,y) \ n) < C{xy \ n) + 0(l) 
(*), and we would obtain the difficult direction in the Chain Rule, C{xy \ n) > C{x \ n) + C(y | x), 
modulo the small additive term. However, in order to have that term small enough for the desired level of 
tightness in the Chain Rule, the randomness extraction needs to be finely tuned (this is the most technical 
part of the proof) depending on some attributes of x and y. Thus, E will not be fully computable, as 
required by the standard definition of a Kolmogorov extractor. Instead, E needs a few bits of information 
about its inputs, and even if this weakens the inequality (*), it still allows us to achieve the desired level 
of tightness in the Chain Rule. 

To obtain the extractors E that require a small amount of non-uniform information about the source 
(see Fact [3] and Fact H|), the use of Kolmogorov extractors is quite direct. The goal here is to show that 
for each x, it is enough to have a short string a^. such that E{x,ax) has randomness rate 1 and contains 
almost all the randomness of x. The solution is based on the fact that from x and a short string that 
is random even conditioned by x, one can extract almost all the randomness of x. This is similar to the 
well-studied case of seeded extractors, with the remark that we can have shorter seeds because requiring 
that the output has randomness rate equal to 1 is weaker than requiring that the output is statistically 
close to the uniform distribution (as stipulated in the definition of seeded extractors). Then we take ax to 
be such a short seed. The above fact is obtained via an elementary use of the probabilistic method. We 
first identify a combinatorial object, called a balanced table, that characterizes a Kolmogorov extractor, 
in the sense that the table of a Kolmogorov extractor must satisfy the combinatorial constraints of a 
balanced table. We show (with the probabilistic method) that such an object exists with a seed of length 
a;(log(n/m)), where n is the length of x and m is the Kolmogorov complexity of x. This establishes 
Fact El Since the function E from Fact [3] is obtained via the probabilistic method, we cannot claim any 
complexity bound for it. To obtain the Kolmogorov extractor in Fact HI which is computed by polynomial- 
size circuits, we derandomize the construction from Fact El using a method of Musatov [MuslOj . The key 
observation is that the combinatorial constraints of a balanced table can be checked by constant-depth 
circuits of relatively small size. The argument goes as follows: (a) these constraints require that in all 
sufficiently large rectangles of the table no element appears too many times; (b) thus one needs to count 
the occurrence of each element in every sufficiently large rectangle of the table; (c) by a well-known result 
of Ajtai |Ajt93 , this operation can be done with sufficient accuracy by constant-depth circuits with 



relatively small size. Therefore, we can use the Nisan-Wigderson ( |NW94j ) pseudo-random generator 
NW-gen that fools bounded-size constant-depth circuits and has seeds of size polylogaritmic in the size of 
the output. Since balanced tables with the required parameters are abundant, we infer that there exists 
a seed s so that NW-gen(s) is a balanced table with the required parameters. A balanced table is an 
object of size exponential in n, which implies that the seed s has size polynomial in n. Moreover, the 
Nisan-Wigderson pseudo-random generator has the property that each bit of the output can be calculated 
separately in time polynomial in the length of the seed. This implies that the Kolmogorov extractor whose 
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table is NW-gen(s) can be computed by a polynomial-sized circuit that has s hard-wired in its circuitry. 

2 Preliminaries 

2.1 Notation and basic facts on Kolmogorov complexity 

The Kolmogorov complexity of a string x is the length of the shortest effective description of x. There are 
several versions of this notion. We use here the plain complexity, denoted C(x), and also the conditional 
plain complexity of a string x given a string y, denoted C(x | y), which is the length of the shortest effective 
description of x given y. The formal definitions are as follows. We work over the binary alphabet {0, 1}. 
A string is an element of {0, 1}*. If x is a string, denotes its length. Let M be a Turing machine 
that takes two input strings and outputs one string. For any strings x and y, define the Kolmogorov 
complexity of x conditioned by y with respect to M, as Cm{x \ y) = min{|p| | M{p,y) = x}. There is a 
universal Turing machine U with the following property: For every machine M there is a constant cm 
such that for all x, Cu{x \ y) < Cm{x \ y) + cm- We fix such a universal machine U and dropping the 
subscript, we write C(x | y) instead of Cu{x \ y). We also write C{x) instead of C{x \ A) (where A is the 
empty string). The randomness rate of a string x is defined as rate(x) = ^j^- If ?^ is a natural number, 
C{n) denotes the Kolmogorov complexity of the binary representation of n. For two n-bit strings x and 
y, the information in x about y is denoted I{x : y) and is defined as I{x : y) = C{y \ n) — C{y \ x). 

In this paper, the constant hidden in the O(-) notation only depends on the universal Turing machine. 

For all n and k < n, 

Strings xi,X2, ■ ■ ■ ,Xk can be encoded in a self-delimiting way (i.e., an encoding from which each string 
can be retrieved) using + |x2| + . . . + |xfc| + 2 log |x2| + . . . + 2 log \xk \ + 0{k) bits. For example, xi 
and X2 can be encoded as (6m(|x2|)01xiX2, where bin{n) is the binary encoding of the natural number n 
and, for a string u = ui . . . Um, u is the string uiui . . . UmUm (i-e., the string u with its bits doubled). 

All the Kolmogorov extractors in this paper are ensembles of functions / = (/n)nGN of type fn : 
{0, !}"• X {0,1}'^'^"'^ {0,1}™'^"^. For readability, we usually drop the subscript and the expression 
"ensemble / : {0, 1}" x {0, 1}'= ^ {0, 1}™" is a substitute for "ensemble / = (/n)n6N) where for every n, 
/„ : {0, 1}" X {0, l}'^^") ^ {0, ." 

For any n E N, [n] denotes the set {1,2,..., n}. 

2.2 Approximate counting via polynomial-size constant-depth circuits 

In the derandomization argument used in the proof of Theorem l4.81 we need to count with constant-depth 
polynomial-size circuits. Ajtai |Ajt93| has shown that this can be done with sufficient precision. 

Theorem 2.1 (Ajtai's approximate counting with polynomial size constant- depth circuits.) There exists 
a uniform family of circuits {GnlneN; of polynomial size and constant depth, such that for every n, for 
every x S {0, 1}", for every a € {0, . . . , n — 1}, and for every e > 0, 

• // the number of I's in x is < {1 — e)a, then G„(x, a, 1/e) = 1, 

• If the number of I's in x is > {1 + e)a, then Gn{x, a, 1/e) = 0. 

We do not need the full strength (namely, the uniformity of Gn) of this theorem; the required level of 
accuracy (just e > 0) can be achieved by non-uniform polynomial-size circuits of depth d = 3 (with a 
much easier proof, see [ViolO] ). 
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2.3 Pseudo-random generator fooling bounded-size constant-depth circuits 

The derandomization in the proof of Theorem 14.81 is done using the Nisan-Wigderson pseudo-random 
generator that "fools" constant-depth circuits |NW94| . Typically, it is required that the circuit to be 
fooled has polynomial size, but the proof works for circuits of size 2"" for some small constant a > 0. 

Theorem 2.2 (Nisan-Wigderson pseudo random generator.) For every constant d there exists a constant 
a > with the following property. There exists a function NW-gen : 
for any circuit G of size 2"" and depth d, 

\P^oh ^^^^^^^o(io,2^+^u)[G{NW-gen{s)) = 1] - Prob,e|o,i}n [G(z) = 1]| < 1/100. 

Moreover, there is a procedure that on inputs (n, i, s) produces the i-th hit of NW-gen{s) in time 
poly(logn). 

3 Symmetry of information for random strings 

We start by proving the new form of the Chain Rule in Fact [2j The formal statement is as follows: 

Theorem 3.1 For all strings x G {0, 1}" and y S {0, 1}" with C{x \ n) > 13 log n + I{x : y) + 0(1) and 
C{y I n) > 71ogn + /(x : y) + 0(1), 

C {xy \n) > C{x \ n) + C{y \ x) — log I{x : y) 

-0(C7(2)(x I n) + C(2)(y I n) + C^^\y \ x)). 

Proof. Let x and y be as in the statement of the theorem. We introduce some notation: 

• tx = C{x I n), 

• ty = C{y I n), 

• = C{y I x), 

. d = 2{C{tx I n) + C{ty i n) + C{ty,x \ n)) + I{x : y) + 0(1), 

(note that d = 0{C^^\x \ n) + C^^\y \ n) + C^^\y \ x)) + I{x : y)), 

• kx = tx - 2{C{tx ! n) + C{ty I n) + C{ty^^ \ n)) - 0(1), 

• ky =ty - 0(1), 

• m = kx + ky — log d — 0{1), 

where the constant 0(1) only depends on the universal machine and will be chosen later. We also 
denote = 2^"' , Ky = 2^y , N = , D = 2'^ and M = 2"^. We state two claims, which we prove 
later, that immediately establish the theorem. The first claim shows the existence of a function E : 
{0, 1}" X {0, 1}" {0, 1}™' that satisfies certain combinatorial constraints similar to those of a balanced 
table, whose parameters are tailored for what we need. We view E as an [A^]-by-[A^] table colored with 
colors in [M]. A rectangle Bi x B2, where Bi C [N] and B2 C [N], is the part of the table formed by 
the rows in Bi and the columns in i?2- For A C [M], we say that a cell {u,v) of the table is an j4-cell, if 
E{u,v) e A. 

Claim 3.2 There exists E : {0,1}"' x {0,1}" {0,1}™ such that for every rectangle Bi x B2, with 
\Bi\ > Kx, I-B2I > Ky, and for every A CI {0, 1}™ with \A\ > jj, the number of A-cells in Bi x B2 is 

<2.^4-\B,\-\B2\. 
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Note that given Ti^tx^ty^tyxt Citx \ ^);^(^^ | C(ty^x I one Ccin effectively enumerate the 
tables that satisfy the claim. Let E be the first table that appears in this enumeration. We denote 
the tuple {tx,ty,ty^xiC{tx \ n),C{ty \ n),C{ty^x I by A. Given n, the tuple A can be encoded 
in a self-delimiting way using C{tx \ n) + C{ty \ n) + C{ty^x \ n) + C{C{tx \ n) + C{C{ty \ n)) + 
C{C{ty^x\ n))+21ogC(t^ I n)+2logC{ty\ n)+2logC{ty^x \ n)+2\ogC{C{tx \ n) +2 log C {C {ty \ n)) + 
2logC{C{ty^x I n)) + 0(1) bits, a value which we denote by A. Using C{C{tx \ n)) < logC{tx \ n) and 
the other similar inequalities, we see that for an appropriate choice of the constants, d > X + I{x : y). 

The second claim shows that E extracts almost all the randomness in x and y. 

Claim 3.3 C{E{x, y) \ n,m) > m — d. 

The theorem follows easily from the two claims. Note that the calculation of E[x^ y) requires a description 
of X, y and A bits for the self-delimited description of E. Then 

C{xy I n) > C{E{x, y)\n,m)-X- 0(1) 
>m-d-X-0{l) 
= kx + ky-logd-d - X- 0(1) 
= — : y) 

-6(0(2)(a; I n) + C^'^\y \ n) + C^'^\y \ x)) 

- log(2(C(2) {x\n) + 0(2) {y\n) + O^^) {y \ x)) + I{x : y)) - 0{l) 
>tx + ty- I{x : y) - log I{x : y) 

-0(C(2)(x I n) + 0(2)(y I n) + 0(2)(y I x)), 

where in the last line we have used the fact that 

log(2(C(2)(3; I n) + C<^\y \ n) + C(^\y \ x)) + I{x : y)) 

< log(2(C(2)(x I n) + 0(2)(y I n) + 0(2)(y I x))) +logI(x : y). 

Since tx = C{x \ n) and ty — I{x : y) = C{y \ n) — {C{y \ n) — C{y \ x)) = C{y \ x), the conclusion 
follows. 

It remains to prove the two claims. 

Proof of Claim \3.Si We use the probabilistic method. The details are presented in the Appendix. 

Proof of Claim HOI Suppose that C{E{x,y) \ n,m) < m - d. Let A = {w e {0, 1}™ | C{w \ n,m) < 
m — d + 0(1)}, where the constant 0(1) (depending only on the universal machine) is chosen so that 
1^1 > 2"^-"^ = ^. It also holds that |^| < 2'^-'^+^(i). 

We define By = {v & {0, 1}"" \ C{v \ n) < ty}. For a convenient choice of the constant appearing in 
the definition of ky, it holds that \By\ > Ky. Also note that \By\ < 2*"+^. 

We say that a row u G {0, 1}" is bad, if the number of A-cells in the {u} x By rectangle of E is 
> 2 • -1^ • I R I 

The number of bad rows is bounded by Kx (otherwise, E would not satisfy the requirement in 
Claim [32] for the rectangle formed by the bad rows and By). Note that, given n,tx,ty,ty^x,C{tx \ 
n), C{ty I n), C{ty^x I n), one can enumerate the set of bad rows. Therefore a bad row u can be described 
by its rank in an enumeration of the set of bad rows, and by the information A required to run this 
enumeration. Therefore if u is a bad row, then 

C{u\n) <kx + X 

< tx. 

Since C{x \ n) = tx, x \s a, good row. Therefore the number of A-cells in {x} x By is 

<^ 2 • • I R I 

< 2 . 2!^^^^ . 2*^+1 

^ 2ty-d+0{l) 
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By our assumption, (x, y) is an A-cell, and, obviously, it is in the {x} x By rectangle. Given x, y can 
be described by the rank of {x, y) in an enumeration of A-cells in {x} x By and by the information A 
required to run this enumeration. 
Thus, 

C{y\x) <ty-d + 0{l) + X 
<ty- I{x : y) 

= C{y I n) - (C{y \ n) - C{y j x)) 
= C{y\x), 

which is a contradiction. | 

Prom Theorem 13. H it is easy to derive the strong Symmetry of Information relation for random strings 
stated in Fact [TJ The formal statement is as follows. 

Theorem 3.4 For every constant c > 0, there exists a constant c' > with the following property: 
For every n € N, for every c-random string x € {0,1}" and every c-random string y € {0,1}", if y is 
c-random conditioned by x, then x is d -random conditioned by y. 

Proof. Note that \iC{x\n)>n- c and C{y\x)>n- c, then C^'^\x \ n) = 0(1), C^'^\y \ n) = 0(1) 
and C(2)(a; | n) = 0(1). Then I{x : y) = 0(1). 

Therefore, from Theorem 13. H we obtain C{xy \ n) > 2n — 0(1). We also have C{xy \ n) < 
C{y I n) + C{x \ y) + 2C{C{y \ n)) + 0(1) = C{y \ n) + C{x \ y) + 0(1). Thus, C{x \ y) > 
2n-C{y\ n) -0(1) = n-0{l). | 



4 Randomness extraction with small advice 

In this section we study the amount of non-uniformity that is necessary for randomness extraction from 
a single source of randomness. 

Vereshchagin and Vyugin |VV02j show the limitations of what can be extracted with a bounded 
quantity of advice. To state their result, let us fix n = length of the source, h = number of bits of advice 
that is allowed, and m = the number of extracted bits. Let H = 2^^^ — 1. 

Theorem 4.1 ((VV02]) There exists a string x G {0,1}" with C{x) > n - i71og(2'" + 1) « n - 
Hm such that any string z £ {0,1}"^ with C{z \ x) < h has complexity C{z) < h + logn + logm + 
0(log log n, log log m) . 

The next theorem, a consequence of Theorem l4.1l shows that no Kolmogorov extractor for sources with 
randomness rate a and that uses h bits of advice about the source can output strings with randomness 
rate larger than 1 — (1 — (t)/H. 

Theorem 4.2 Assume that the parameters m, h, a are computable from n and satisfy the following rela- 
tions: 0<a<l,h>0,0<m<n, m = a;(logn + h). 

Let f : {0,1}" X {0,1}'^ — > {0,1}™ be a computable ensemble of functions such that for every x G 
{0,1}" withC{x) >(T-n, there exists a string ax such that C {f {x , ax)) > (1 — e)-m. Then e > — o(l). 

Proof. Let m' = min([i^ • n\,m). Note that m' > [^^f- ■ rn]. Let x be the string guaranteed 
by the Vereshchagin- Vyugin Theorem 14.11 for the parameters n,h -\- c,m', where c is a constant that 
will be specified later. Note that C{x) > n — H ■ m' > a ■ n. By assumption there is a string ax such 
that C{f{x,ax)) > (1 — e)m. Let z be the prefix of length m' of f{x,ax)- Note that C{f{x,ax)) < 
C{z) -\- (m — m') -\- 2 logm -|- 0(1), which implies that C{z) > (1 — e)m — m + m' — 2 logm — 0(1) > 
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We also have C{z \ x) < \ax\ + c = h + c, for some constant c. It follows from Theorem 14.11 that 
C{z) < /i + logn + logm' + 0(loglogn,loglogm'). So, ^^^^ m — em — 21ogm- — 0(1) </i + logn + 
0(loglogn, loglogm'), which implies that e > — = — o{l). I 

We move to showing the positive results in Fact E] and Fact U] regarding randomness extraction with 
small advice that complement the negative result in Theorem 14.21 The constructions use the parameters 
n,ni,m,k,6 and d. We denote N = 2",7Vi = 2"i , M = 2™, A = 2^ and D = 2'^. We identify in the 
natural way a function E : {0, 1}" x {0, 1}"^ — )■ {0, l}'" with an [N] x [A'^i] table colored with colors from 
[M]. For A C [M], we say that an {u,v) cell of the table is an A-cell if E{u,v) G A. The reader might 
find helpful to consult the proof plan presented in the Introduction. As explained there the notion of a 
balanced table plays an important role. 

Definition 4.3 A table E : [N] x [Ni] [M] is {K, D, A) -balanced if for any B C [N] with \B\ > K, 
for any A C [M] with > j), it holds that 

\A-cells in B X [Ni]\ ^ \A\ 

\B\VWi - ' W' 

The following lemma shows that a balanced table is a good Kolmogorov extractor. 

Lemma 4.4 Let E : [N] x [Ni] [M] be a {K, D, A) -balanced table and d = 5 -\- 0(1). Suppose 
C{E I n) = 0(1) and ni,k,d, and 5 are computable from n. Let {x,y) G [N] x [Ni] be such that 
C{x \ n) >k + 0(1) and C{y \ x) > ni. Let z = E{x, y). Then C{z \ m) > m — d. 

(Note: 0(1) means that there exist constants, depending only on the universal machine, for which the 
statements hold.) 

Proof. The proof is similar to the proof of Claim [331 We sketch the argument. Suppose 0(2 | m) < m—d. 

Let A = {w £ {0,1}"* I C{w I m) < m — d + 0(1)}, where the constant 0(1) is chosen so that 
1^1 > 2"^-^. Also note that |yl| < 2'"-°'+'^(i) . 

We say that a row v is bad if the number ^-cells in the {w} x [A^i] rectangle oi E \s > A ■ ■ Ni. 
The number of bad rows is at most K, because the table E is {K, D, A)-balanced. Therefore a bad row 
V is described by the information needed to enumerate the bad rows (and this information is derivable 
from n) and from its rank in the enumeration of bad rows. So, if v is bad, C{v \ n) < k -\- 0(1). 

Since 0(x | n) > A: + 0(1), it follows that x is good. Therefore, the number of A-cells in the {x} x [Ni] 
rectangle of ^ is < A • ^ • iVi = 2'^-<^+"i+0(i). 

Note that, by our assumption, the cell (x, y) is an ^-cell. Then the string y, given x, can be described 
by the rank of {x,y) among the ^-cells in the {x} x [Ni] rectangle of E. 

So, C{y \ x) < 5 — d + ni + 0(1) and the right hand side is less than ni for an appropriate choice of 
the constant 0(1) in the relation between d and 5. We obtain that C{y \ x) < ni, contradiction. | 

The next lemma establishes the parameters for which balanced tables exist. 

Lemma 4.5 Suppose the parameters satisfy the following relations: D = 0(A), n/S = o{Ni), and M = 
o{5- K ■ Ni). Then there exists a table E : [N] x [Ni] [M] that is {K, D, A)-balanced. 

Proof. The proof is by the probabilistic method and is presented in the Appendix. | 
We can now prove Fact [3l The formal statement is as follows. 

Theorem 4.6 Parameters: Let m[n) and h{n) be computable functions such that m{n) < n for all n 
and h{n) = u:{\og:^). 

There exists a computable function E : {0, 1}" x {0, 1}^(") — {0, Ij^-C") ^ such that for every x € {0, 1}" 
with C{x I n) > m{n), there exists ax G {0, 1}'*(") such that C{E{x,ax)\m) > m{n) — o{m{n)). 
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Proof. We take 5 = ^o.shin) i d = 6 + c, where c is the constant from Lemma 14.41 ni = h{n). 

By Lemma [331 there exists a table E : [N] x [Ni] [M] that is (K, D, A)-balanced, and by brute 
force one can build such a table from n. Thus we obtain such a table E with C{E \ n) = 0(1). We 
take ax to be a string in {0,1}'^^"^ such that C{ax \ x) > h{n). Using Lemma 14.41 we obtain that 
C{E{x, Ux) I m) > m(n) -d = m{n) - ^oZ^n) - c = m - o{m). | 

Our next goal is to derandomize the construction in Theorem 14.61 As explained in the Introduction 
the key observation is that checking if a table is balanced can be done, in an approximate sense, by 
constant-depth circuits with relatively small size. 

Lemma 4.7 The parameters ni,m,k,d, and 6 are positive integers computable from n in polynomial 
time. We assume k < n,m < k,d < n. 

There exists a circuit G of size poly{N^) and constant depth such that for any table E : [N] x [A''i] — )• 

m, 

(a) ifG{E) = I, then E is {K, D,imA)-balanced, 

(b) if E is {K,D, A) -balanced, then G{E) = 1. 

Proof. Let a = (1/0. 99) A • 1/D ■ K ■ Ni. Let us fix for the moment a set of rows B C [N] of size 
\B\ = K and a set of colors A C [M] of size |^| = M/D. Let xb,a be a binary string indicating which 
cells in the B x [A'^i] rectangle of E are A-colored. Formally, xb,a is the string of length K ■ Ni, whose 
(i,j)-th bit is 1 if the cell {i,j) in the rectangle B x [A'^i] of E is an ^-cell and if it is not. 

By Ajtai's Theorem 12.11 there exists a polynomial-size constant-depth circuit G' (which does not 
depend on B and A) with a hardwired and such that 

• G'{xb,a) = 1 if the number of yl-cells in i? x [Ni] is at most (1 — 0.01) • a, and 

• G'{xb,a) = if the number of yl-cells in i? x [A^i] is at least (1 + 0.01) • a. 

Now we describe the circuit G. 

The circuit G on input an encoding of the table E (having length N ■ Ni ■ m) computes in constant 
depth a string xb,a for every B C [N] with \B\ = K and for every A C [M] with A = M/D. There 
are (^) {^^/d) ~ V^^y{^^) such strings Xb,a- Each such string xb,a is the input of a copy of G' . The 
output gates of all the copies of G' are connected to an AND gate, which is the output gate. 

If G{E) = 1, then G'{xb,a) = 1 for all -B's and ^'s as above. This implies that for all B C [N] with 
\B\> K and all A C [M] of size > M/D, the number of ^-cells in the B x [A^i] rectangle of E is at most 
(1 + 0.01)a < (1.03) ■ A-{l/D)-K- Ni, i.e., E is {K, D, 1.03A)-balanced. 

In the other direction, if E is (.fC, -D, A)-balanced then for all B C [N] with \B\ = K and for all 
A C [M] with A = M/D, the number of ^-cells in S x [Ni] is at most A ■ {1/ D) ■ K ■ Ni = {I - 0.01)a, 
which implies that G{E) = 1. | 

We next prove Fact [H The formal statement is as follows. 

Theorem 4.8 Parameters: Let m{n) and h{n) be polynomial- computable functions such that m[n) < 
0.99 • ONW ■ n and h{n) = w(log(^^)). 

There exists a function E : {0, 1}" x {0, 1}^(") — )• {0, 1}™("), computable by an effectively constructible 
circuit having polynomial size and the following property: For every x G {0, 1}" with C{x \ n) > m{n) + 
0{1), there exists a string ax € {0, l}''^") such that C{E{x,ax) \ m) > m — o{m). 

Proof. Let k — 771(71), 5 — 20.5h(ra) 1 d — 5 -\- c-\- log 1.03 (where c is the constant from Lemma I4.4p . and 
ui = h{n). 

Let G be the circuit promised by Lemma 14.71 for these parameters. Let dAjtai be the depth of the 
circuit G and let a^w be the constant corresponding to dAjtai in Theorem 12. 2[ 
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Let iV = iV • A^i • m. This is the size of an encoding of a table E : [N] x [Ni] [M]. Let 
NW-gen : {0,1}^°^ Ajtai+Gf^j^-j _^ {0,1}^ be the Nisan-Wigderson pseudo-random generator given by 
Theorem 12.21 for the depth parameter equal to dAjtai- Note that poly(A''^) < 2^"^^^ , where poly(A^^) 
is the bound from Lemma 14.71 for the size of the circuit G. 

The probabilistic argument in Lemma 14.51 can be modified to show that among the tables of type 
E : [N] X [A^i] — )• [M] the fraction of those which are {K, D, A)-balanced is at least 0.51. Since G accepts 
all such tables, 

Since the circuit G has depth equal to dAjtai and size bounded by 2^"^^ , it follows that if we replace a 
random E G {0, 1}^ by NW-gen(s) for a random seed s € {0, l}'°s we obtain 

We only need the fact that there exists a string s G {0, 1}'°^ ^^^""^ such that NW-gen(s) is a table 
E : [N] X [A'^i] — )■ [M] that is {K,D, 1.03A)-balanced. We fix such an s that is computable from n (say, 
the smallest s that has the property) and the corresponding table E produced by the Nisan-Wigderson 
pseudo-random generator on seed s. 

Let us consider x S {0,1}" with C{x \ n) > k and G {0,1}"^ with C(a^ | y) > ni. Since E is 
{K^ D, 1.03A)-balanced, it follows from Lemma 14.41 that C{E{x, ax) \ m) > m — d = m — o{m). 

Now, let us view E (which is NW-gen(s)) as a function E : {0, 1}"' x {0, 1}"^ {0, 1}'". From the 
properties (i.e., the "Moreover ..." in Theorem 12. 2p of the Nisan-Wigderson pseudo-random generator, it 
follows that this function can be computed by a polynomial-size circuit which has s hardwired. Since 
s is also computable from n, one can compute a description of the circuit, i.e., the circuit is effectively 
constructible. I 
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A Appendix 



Proof of Claim [3721 

We use the probabilistic method. It is enough to show the assertion for all Bi, B2 and A having sizes 
exactly K^, Ky, and respectively Let us consider a random function E : {0, 1}" x {0, !}"■ — {0, l}*". 
Fix Bi,B2 and A, satisfying the above requirement on their sizes. By the Chernoff's bound, 

Prob[|yl-cells in 5i x ^a] > 2 ■ ^ • • 1^2 1] 

< g-{l/3)(|A|/M)|Bi|[B2| ^ ^-{l/2.)(l/D)K^Ky^ 

The sets Si, B2, and ^ can be chosen in (£)-(#J-(m%) < iV^-A^^--(eZ))^^/^ = 1°^+^. 1-^ 
ways. Since tx > 13 log n + /(x : y) + 0(1) and d < 61ogn + I{x : y) + 0(1), it follows that 
tx ^ d + 71ogn + 0(1) and from here kx > d + logn + 0(1). Since ty > 71ogn + I{x : y) + 0(1), it 
follows that ky > d + logn + 0(1). It can be easily checked that, given these bounds for kx and ky and 
for an appropriate choice of the constant in the definition of d, 

^-{l/?,){l/D)K^Ky . ^K^\nN+Ky\nN+(MID)+{M/D)\nD ^ ^ 

Thus, the probability that a random E satisfies the requirements is less than 1, which implies that there 
exists an E satisfying the claim. | 

Proof of Lemma 14.51 . 

The proof is by the probabilistic method. Consider a random function E : [N] x [A'^i] — [M]. We 
evaluate the probability that E fails to be (X, D, A)-balanced. Note that if E fails to be {K, D, A)- 
balanced, then there exists a set B C [N] of size exactly K and a set A C [M] of size exactly M/D such 
that the fraction of A cells in the B x [A^i] rectangle of E is greater than A • l^dj/M. Let us call this 
latter event S. We show that the probability of S is less than 1. Fix B C [N] of size K and A C [M] of 
size M/D. For a fixed (x,y) £ B x [Ni], Prob[£'(x,y) G A] = \A\/M. The expected number of A-cells in 
B X [Ni] is n= \B\ ■ Ni ■ \A\/M. Let A' = A - 1. 

We use the following version of the Chernoff bound. If X is a sum of independent Bernoulli random 
variables, and the expected value E[X] = /i, then Prob[X > (1 + A)fi] < e-^(i°(^/3))Mj3| 

Using these Chernoff bounds, 

Prob[|A-cells in B X [iVi]| > (1 + A>] < e-^'(i^(^'/3))M_ 

The set B can be chosen in (^j < N ways. The set A can be chosen in (J^^) < (eD)^^/^ ways. It 
follows that the probability of S is bounded by 

A^-^ • {eDf'^/^ ■ e-^'(i'i('^'/3))-i^-iVr(i/o)^ 
which, taking into account the relations between parameters, is less than 1. | 



The standard Chernoff inequahty Prob(X > (1 + A)/i] < ( (i-|-a)(i+a) ) is presented in many textbooks. It can be 



checked easily that (,^^")(i+a) < e""^'"*^/^'. 
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